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ABSTRACT 



The present invention provides a method and apparatus for 
linking real-time data with audiovisual content to enable a 
user to make selections, manipulate data, and to execute 
functions interactively through an audiovisual display unit 
based upon audiovisual content being displayed at the time. 
Audiovisual content is synchronized with the available 
interactive functions so that the user has the abihty to 
interact with and perform functions related to the audiovi- 
sual content being shown at the time. In a preferred 
embodiment, actionable events in audiovisual programs are 
synchronized with associated overlay functions and inter- 
face data. A method is provided to make overlay functions 
available to the user through an audiovisual display during 
the actionable events that occur in a program being viewed 
by the user. A user interface for each overlay function that 
may be performed, which user interface may or may not be 
visible to the user, is overlaid on and synchronized with the 
program being shown. Interaction with selections may be 
made using any of several known methods such as a screen 
menu or a cursor controlled by a remote control device. 

9 Claims, 5 Drawing Sheets 
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VISUAL OBJECT PRESENT WITHIN LIVE 
PROGRAMMING AS AN ACTIONABLE 
EVENT FOR USER SELECTION OF 
ALTERNATE PROGRAMMING WHEREIN 
THE ACTIONABLE EVENT IS SELECTED 
BY HUMAN OPERATOR AT A HEAD END 
FOR DISTRIBUTED DATA AND 
PROGRAMMING 

BACKGROUND OF THE INVENTION 

1. Field of the Invention 

The present invention relates to the field of interactive 
audiovisual systems. More particularly, the present inven- 
tion relates to a method and apparatus for linking real-time 
data with audiovisual content to enable a user to make 
selections, manipulate data, and to execute functions by 
interacting with the audiovisual content being shown at the 
time. 

2. Art Background 

With the development of computer technology, interactive 
systems are possible that allow users to retrieve data and 
transact business through various means, llirough the 
telephone, without speaking to a live operator, users can 
retrieve various types of information, order pay-per-view 
cable programs, activate credit cards, obtain customer sup- 
port for products, order movie tickets and listen to and buy 
musical selections. Using the Internet and commercial 
online services, users are able to read news reports, view 
photographs and video clips, listen to sound files, send 
electronic mail, converse and videoconference in real time 
with users in remote locations, view sports statistics, buy 
products, generate maps, and retrieve archived data. 

Often a user may decide to use the telephone or an online 
service to perform such functions based upon programming 
viewed on television. For example, a user may view a music 
video on television and then decide to use an automated 
telephone service or the Internet to order an album by the 
artist shown. While viewing a football game, a user might be 
prompted to use the Internet to retrieve statistics for a certain 
player or team. Users desire to have the ability to perform 
these functions and new functions using the television 
instead of having to use a different device such as a 
computer or telephone. 

The advances in digitization and digital compression of 
data, improvements in transmission media such as fiber 
optics, and the combination of computer technology with 
television and audiovisual systems has fostered the devel- 
opment of interactive audiovisual systems. In interactive 
audiovisual systems, broad bandwidth permits two-way 
communication, or "interactivity," between the user and a 
source provider. Such systems allow the user, utilizing a 
remote control device in conjunction with a set-top box 
coupled to a cable television, satellite direct broadcast or 
other communication system, to select a desired movie or 
program to be viewed, request services and perform other 
functions such as those that may be performed today using 
telephones and computer networks. Such systems generally 
require the user to access such functions by using channels 
separate from channels on which programming is shown 
and/or by using menus to access such functions. Often a 
cursor system is used to allow the user to perform functions 
using a remote control. 

While such systems enable selection of programs and 
other functions through the television, the selection is not 
enabled as a function of the content appearing on the 
television screen. The user must enter a certain mode or 
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otherwise enable a function which is always available to the 
user upon request. The present invention provides a method 
for enabling functions only at specified times based upon the 
content of audiovisual programming being shown to the user 

5 at the time. Furthermore, the functions enabled can be made 
to depend upon the content of the present audiovisual 
programming, so that the functions can be tailored to spe- 
cific moments or events in an audiovisual presentation. The 
present invention presents the user with the ability to interact 

10 with the programming by selecting items appearing in the 
programming shown on the screen with a cursor, by making 
selections when prompted verbally by an on-screen host, or 
by other means. It will be understood by those skilled in the 
art that the present invention may be used for a variety of 

15 purposes, some of which are described in detail herein. 

SUMMARY OF THE INVENTION 

The present invention provides a method and apparatus 
for enabling a user of an interactive audiovisual system to 

2Q perform functions in real time based upon the content of 
audiovisual programming being shown to the user at the 
time ("overlay functions"). 

The present invention provides a method and apparatus 
for linking real-time data with audiovisual content to enable 

25 a user to make selections, manipulate data, and to execute 
functions interactively through an audiovisual display unit 
based upon audiovisual content being displayed at the 
current time. Put another way, audiovisual content is pre- 
sented in synchronization with selected overlay functions so 

30 that the user has the ability to interact with the audiovisual 
content by performing said overlay functions associated 
with selected events that occur in the programming 
("actionable events"). 
A method and apparatus are provided to make overlay 

35 fii net ions available to the user through the audiovisual 
display during the actionable events that occur in program- 
ming being viewed by the user. A user interface for each 
overlay function that may be performed, which user inter- 
face may or may not be visible to the user, is overlaid on and 

40 synchronized with the programming being shown. Interac- 
tion with selections may be made using any of several 
known methods such as a screen menu or a cursor controlled 
by a remote control device. For example, while viewing a 
movie starring a famous celebrity, the set of overlay func- 

45 tions may include the ability to retrieve in an overlay 
window a list of recent movies starring the celebrity by 
selecting the celebrity's name during the opening credits of 
the movie, or whenever the celebrity appears on the audio- 
visual display. In some instances, a visible cue may exist to 

50 indicate that a function may be performed, such as a visible 
graphical outline appearing on the audiovisual display 
around an item that may be selected. As another example, 
during a football program, a user may be able to retrieve a 
team roster by selecting the name of a team when shown, or 

55 the career statistics of a player by selecting that player. As 
another example, while viewing a television program which 
reviews current movies, a user would have the ability to 
access an Internet World Wide Web site for a given movie 
by making a selection while that movie is being reviewed. 

60 As another example, the present invention may be used to 
implement an audiovisual program guide that allows a user 
to select programming being shown at the time by giving the 
user the ability to select one of several areas on the user's 
television screen, with each of the several areas displaying 

65 a different program that is available to the user at the time. 
In the present invention, a set of overlay functions 
("overlay ftinction set") is associated with each program to 
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be shown to users. The overlay functions could include a 
variety of retrievable information and interactive functions 
that will be available to the user while the user is viewing the 
program. Each program may have unique overlay function 
sets since the overlay functions made available are based 5 
upon the unique content of each program. For each action- 
able event in a program, the operator will determine the 
overlay function set thai will be made available to the user, 
and the characteristics of the interface that will both present 
the availability of the overlay functions to the user and allow lO 
the user to access the overlay functions. Overlay function 
sets and interface data for a plurality of programs are stored 
in memory. When a program having overlay function sets 
and interface data associated with it is sent by the source 
provider to a user (whether requested on demand, shown at is 
a pre-determined time, or otherwise), the overlay function 
sets and interface data for the program, which may also be 
sent by the source provider, arc synchronized with the 
program and loaded into the set-top box. The processor in 
the set-top box reads the overlay function set and interface 20 
data and then both implements the interface and enables the 
overlay functions in real time and in synchronization with 
the associated actionable event in the programming. As the 
user interacts with the program through the synchronized 
interface, the set-top box implements the relevant overlay 25 
functions selected by the user. Alternatively, the set-top box 
may communicate with the source provider as necessary to 
implement certain overlay functions selected by the user. 

BRIEF DESCRIPTION OF THE DRAWINGS 30 

FIG. 1 is a functional block diagram of a head end of an 
interactive television system utilizing the teachings of the 
present invention. 

FIG. 2 is a functional block diagram of a remote user 
location of an interactive audiovisual system utilizing the 
teachings of the present invention. 

FIG. 3 shows one possible implementation of synchroni- 
zation data, 

FIG. 4 is a flowchart of the detailed process performed by 40 
the processing unit in the head end of an interactive audio- 
visual system in one embodiment of the present invention. 

FIG. 5 is a flowchart of the detailed process performed by 
the processing unit in the set-top box in one embodiment of 
the present invention. 

FIG. 6 illustrates the basic protocol fields that facilitate 
the synchronization of overlay function sets and interface 
data with actionable events. 

DETAILED DESCRIPTION OF THE 50 
INVENTION 

1. General Considerations 

In the present case, the operations described herein are 
machine operations performed automatically or in conjunc- 55 
tion with a human operator. Useful machines for performing 
the operations of the present inventions include general 
purpose digital computers, digitally controlled displays, or 
other similar devices such as set-top box control systems, 
video on demand servers and the Uke. The present invention 60 
relates to method steps for operating an interactive audio- 
visual system and processing electrical or other physical 
signals to generate other desired physical signals. 

The present invention also relates to an apparatus for 
performing these operations. ITiis apparatus may be spe- 65 
cially constructed for the required purposes or it may 
comprise general purpose computers selectively activated or 
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reconfigured by computer programs stored in the computers. 
The method steps presented herein are not inherently related 
to any particular computer or other apparatus. In particular, 
various general purpose machines may be used with pro- 
grams in accordance with the teachings herein, or it may 
prove more convenient to construct more specialized appa- 
ratuses to perform the required steps. 

Additionally, no particular programming language has 
been indicated for carrying out the various methods 
described herein. Each designer of a particular computer, 
interactive multimedia system, or other display apparatus 
will be aware of a language which is most suitable for his or 
her immediate purposes. Because the computers, servers, 
processing units, set-top boxes and other devices which may 
be used in practicing the present invention consist of many 
diverse elements, no detailed program listing has been 
provided. It is considered that the operations and other 
procedures described herein and illustrated in the accompa- 
nying drawings are sufliciently disclosed to enable one of 
ordinary skill to practice the present invention. 

In a typical interactive audiovisual system, a head-end is 
coupled to a wire or wireless, analog or digital, communi- 
cation medium such as cable, satellite, microwave, terres- 
trial wave or optical fiber, which is in turn coupled to a 
plurality of users. At the location of each user, a set-top box 
is coupled to the communication medium for receiving 
programming and data from the head end. In some systems 
it may be necessary for the set-top box to send data back to 
the head end via the same communication medium or a 
different communication medium. For example, a satellite 
signal could be used to transmit programming and data from 
the head end to the set-top box, while a wire telephony 
system may be used to send data from the set-top box back 
to the head end. The set-top box is also coupled to a 
television or other audiovisual display unit to enable the 
presentation of programming and data sent from the head 
end. The set-top box may include a processing unit coupled 
to a memory and other electronic modules such as decoders, 
decompressors and audiovisual generators. Computer pro- 
gramming and data files are stored within the memory and 
accessible by the processing unit. The processing unit gen- 
erates and displays a graphical user interface on the user's 
audiovisual display unit, and controls any interactive com- 
munication between the user and the head end. Typically the 
user's system will include a user input device, such as a 
remote control device, for use in conjunction with the set-top 
box to enable the user to make selections, change channels 
and settings, and perform other functions. 

2. General System Configuration 

One embodiment of the present invention may be 
described with reference to FIGS. 1 and 2. FIG. 1 is a 
functional block diagram of a head end of an interactive 
television system utilizing the teachings of the present 
invention. FIG. 2 is a functional block diagram of a remote 
user location of an interactive audiovisual system utilizing 
the teachings of the present invention. In operation, as one 
of the first tasks in implementing this embodiment of the 
present invention, the entire length of each program with 
which users will be able to interact is indexed with sequen- 
tial code data, which sequential code data may correspond to 
the elapsed time of each said program during its presenta- 
tion. For example, the SMPTE time code and the Sony RC 
time code are well-known in the art as forms of sequential 
code data. Once the program has been sequentially coded, 
any actionable event in a program can be uniquely identified 
by the beginning and ending sequential codes during which 
the event occurs. It is to be noted that data other than time 
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may be used to index the program sequentially, such as, for 
exanaple, the number of digital bytes read, the number of 
film frames elapsed, or some other non-time-based means 
for counting. 

As a next step in implementing the system illustrated in 
FIG. 1, the program may be viewed by a human operator, 
before the program is to be transmitted to users, so that the 
operator may determine and record the sequential codes for 
the actionable events. For example, in a certain program it 
may be determined that "silver spaceship appears at sequen- 
tial codes 1:13:43 through 1:14:17" or, in a hosted program 
guide, that the "on-screen host discusses Westerns at sequen- 
tial codes 2:42:02 through 2:48:25." At this point there exists 
a set of data containing unique indicators for each actionable 
event in the programming in the form of sequential code data 
associated with each of those actionable events. 

After the actionable events in a given program have been 
identified using the associated sequential code data, the 
overlay functions that will be made available to the user and 
the interface through which the availabifity of the overlay 
functions will be presented to the user and through which the 
user will access the overlay functions, can be created for the 
program. Instructions to be used by the set-top box to allow 
the user to select overlay functions will be included as part 
of the interface data. As one example of the implementation 
of interface data, user input can be determined by defining 
a screen area of the user's audiovisual display unit which 
may be actively selected by a user, and implementing the 
corresponding overlay function if the user selects the defined 
screen area using a cursor or some other means for indicat- 
ing the selection of a certain screen area. For example, the 
two-dimensional (X, Y) screen coordinates defining a select- 
able area of the screen may be sent to the set-top box as part 
of the interface data. Alternatively, the interface data defin- 
ing a selectable screen area might include the two- 
dimensional coordinates of a single point on the screen and 
the radius of a circle centered at the single point. The set-top 
box may be programmed to generate the areas defined by the 
interface data. In addition to defining the selectable screen 
areas, the interface data may also include visual indicators, 
or overlays, which will appear over the program shown to 
the viewer during the actionable event to indicate where the 
selectable screen areas are located. For example, a certain 
object appearing on the screen may be highlighted in a 
certain transparent color to indicate to the user that the object 
may be selected with a cursor to perform an overlay function 
while the object is highlighted. As another example, the 
interface data may include a graphical, animated or hve- 
action icon that will appear at a certain area on the screen 
while the actionable event is occurring. In this example, the 
location of the icon would coincide with a selectable screen 
area so that the user may implement an associated overlay 
function by selecting the icon. As another example, the 
interface data may be used to implement an audiovisual 
program guide that allows a user to select programming 
being shown at the time. The user's screen may separated 
into two or more screen areas, with each of the areas 
displaying a different program, or an icon or other repre- 
sentation of a different program, that is available to the user 
at the time. As described above, the interface data may 
include the two-dimensional (X,Y) coordinates of each 
screen area, as well as a visual indicator, or overlay, which 
will appear over the selectable screen area coinciding with 
the position of the user's cursor at the time. When the user 
moves the cursor to another selectable screen area, the visual 
indicator would appear over the new selectable screen area, 
which may then be selected by the user with the cursor to 
access the programming represented in that selectable screen 
area. 
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As another example of the implementation of interface 
data, the user might perform a function by pressing a special 
button on a remote control device when a text message 
generated by the set-top box appears at the bottom of the 
5 screen. In such a case, the interface data could include the 
text message to be displayed, or the interface data may 
include instructions to the set-top box to generate the text 
message. 

It is to be understood that the interface need not include 
a graphical overlay to indicate an actionable event. Users 
may be prompted by other means to perform overlay func- 
tions during actionable events. For example, a program may 
include a speaking host (who may or may not appear on 
screen) that tells the user which overlay functions may be 

J 5 performed at the lime as well as how the user may perform 
the functions. Alternatively, the interface data may include 
audio data which is to be heard by the user when overlay 
functions are enabled. 

After the operator has created the interface data and 

20 overlay function set that will be made available to the user 
for a given actionable event, it will be necessary to uniquely 
associate the interface data and overlay fimction set with the 
related actionable event. The interface data and overlay 
function set for a given actionable event may be uniquely 

25 associated with the actionable event by using the same 
sequential code data to identify the actionable event, the 
related overlay function set and the related interface data. 
This may be accomplished by assigning the sequential code 
data for the actionable event to the related interface data and 

30 overlay function set. By assigning sequential code data to 
the interface data and overlay function set that matches the 
sequential code data of the related actionable event, the 
interface data and overlay function set can be retrieved from 
memory at the proper time and synchronized with the related 

35 actionable event by comparing the sequential code data 
assigned to the interface data and overlay function set with 
the current sequential code data during the presentation of 
the program to the user. Accordingly, it is understood that for 
each sequentially coded program, synchronization data is 

40 created by a human operator, which data contains the 
sequential code data for the actionable events in the pro- 
gramming and the location in memory of interface data and 
overlay function set to be retrieved and presented to the user 
when the sequential code data for each actionable event is 

45 encountered. Those skilled in the art will understand that the 
synchronization data may take the form of a look-up table, 
FIG. 3 shows one way in which synchronization data to 
be accessed by the processing unit (12) may be structured. 
In FIG. 3, the synchronization data is depicted as a table of 

50 addresses of the calls for the overlay function set and 
interface data associated with each actionable event in a 
program. It should be understood that additional attributes 
may be included in the synchronization data shown in FIG. 
3, such as a subset of user set-top box addresses for which 

55 a given actionable event will or will not be actionable. For 
example, if a certain group of users are not authorized to 
select overlay functions for a given event, the synchroniza- 
tion data may contain the set-top box addresses for those 
unauthorized users so that the processing unit can be 

60 instructed not to send the overlay function set and interface 
data for the actionable event to the unauthorized users. 

Referring now to head end (2) shown in FIG. 1, the first 
embodiment of the present invention uses sequentially 
coded programming received at the head end (2) by various 

65 means including transmission via satellite (4) from a remote 
location, or such sequentially coded programming may be 
stored in signal storage (6) at the head end. It is to be 
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understood that sequentially coded programming may be lay function sets and interface data with actionable events as 
stored in a variety of formats, including but not limited to disclosed in the present invention. FIGS. 6, 6a, 6b and 6c 
MPEG format, DVD, videotape, or other formats. A illustrate basic protocol fields, to be employed in a transport 
switcher (8) is provided which switches between the pro- layer, that may be used to facilitate the synchronization 
gramming sources depending upon which is desired to be 5 process. The signal combiner structures the combined signal 
transmitted to the user. It is understood by those skilled in as shown in HG. 6. llie sequential code data for a given 
the art that the switcher may take many forms. The switcher actionable event may be sent first, followed by the overlay 
transmits the sequentially coded program to a signal com- function set for the actionable event, followed by the inter- 
bine r (10). It is understood that the signal combiner is face data for the actionable event. As shown in FIG. 6a, the 
well-known in the art and may be a multiplexer, a line 10 sequential code data for the actionable event may consist of 
inserter or other component. the start and end codes for the actionable event. As shown 

The sequentially coded program is also transmitted to a in FIG. 6b, the overlay function set may consist of the calls 
processing unit (12), which may include components such as to be made by processing units to implement each available 
random access memory, processors, coprocessors, and inter- overlay function. As shown in FIG. 6c, the interface data 
face drivers. FIG. 4 illustrates the detailed process per- 15 may consists of screen indicator attributes such as polygon 
formed by the processing unit (12) in the present embodi- shape, location, format, style and transitions, 
ment. At block (60), the processing unit (12) retrieves the The apparatus of the remote user location is shown in 
synchronization data for the program from memory (14). detail in FIG. 2. The combined signal containing the 
The synchronization data is predetermined and may be program, overlay function set and interface data is received 
stored in memory (14) by a human operator or downloaded 20 at a receiving unit, which may be a set-top box (34) as shown 
tomemory automatically from the source of the sequentially in FIG. 2, by a network interface (36) which sends the 
coded program with which the synchronization data is combined signal to a decoder (38) which separates the 
associated, prior to the transmission of the program. At program from the overlay function set and interface data 
block (62), the processing unit (12) reads the sequential code sent from the head end (2). The decoder sends the program 
data associated with the sequentially coded program. At 25 to a signal combiner (50) which sends the program to the 
block (64), the processing unit (12) references the sequential audiovisual display unit (52) for presentation to the user, 
code data to the synchronization data. At block (66), the Simultaneously, the decoder sends the synchronized inter- 
processing unit (12) determines whether an overlay function face data and overlay function set to a processing unit (40). 
set and interface data exist for the current sequential code FIG. 5 illustrates the detailed process performed by the 
data (i.e., whether there is an actionable event occurring in 30 processing unit (40) in the present embodiment. At block 
the programming). At block (68), if an overlay function set (72), the processing unit (40) receives the interface data and 
and interface data exist for the sequential code data read by overlay function set. At block (74), the precessing unit (40) 
the processing unit (12), the processing unit (12) retrieves reads and formats the interface data. At block (76), the 
the interface data from memory (16) and the overlay func- processing unit (40) implements the user interface in accor- 
tion set from memory (18). If an overlay function set and 35 dance with the instructions contained within the interface 
interface data do not exist for the sequential code data being data, including sending visual, audible, textual, and other 
read, the processing unit prepares itself to read the next material to the signal combiner (50) for presentation to the 
sequential code data. The overlay function set and interface user in synchronization with the actionable event being sent 
data is predetermined and may be stored in memories (16 to the audiovisual display unit (52). At block (78), the 
and 18) by a human operator or downloaded to memory 40 processing unit (40) simultaneously reads the overlay func- 
automatically from the source of the sequentially coded tion set and enables overlay functions in synchronization 
program with which the synchronization data is associated, with the actionable event being sent to the audiovisual 
prior to the transmission of the program. At block (70), after display unit (52) so that the user is able to implement the 
retrieving the interface data and the overlay function set various overlay functions. The user may implement those 
from memory, the processing unit (12) sends the overlay 45 functions by making selections using a remote control 
function set and interface data to the signal combiner (10), device (48) which transmits input signals to a control 
which combines the overlay function set and interface data interface (46) coupled to the processing unit (40). At block 
with the sequentially coded program so that the combined (80), the processing unit (40) determines whether the user 
signal output by the signal combiner (10) contains the has selected an overlay function. At block (82), if the user 
interface data and the overlay function set synchronized with 50 has selected an overlay function, the processing unit (40) 
the actionable event identified by the sequential code data causes the selected overlay function to be performed. One or 
read by the processing unit (12). The signal combiner (10) more of the overlay functions selected by the user may be 
sends the combined signal to the head end transmission performed in the set-top box (34) by the processing unit 
equipment (22) for transmission to one or more remote user (40), which may include components such as interface 
locations (as shown in FIG. 2) via communication medium 55 drivers, processors, and co-processors and may include 
(30). It is understood that head end transmission equipment random access memory (42) capable of running applications 
may take various forms and is well-known in the art. It that may be accessible as part of the overlay functions 
should further be understood that, in this embodiment, the available to the user. Such applications may be pre-loaded in 
synchronization of the interface data and the overlay func- the set-top box or downloaded from the head end as needed, 
tion set with the associated actionable event occurs in the 60 Alternatively, one or more of the overlay functions selected 
head end (2). Alternatively, such synchronization may occur by the user may be performed at the head end (2) or other 
at a remote location from which the synchronized interface remote location. Signals received from the user at the control 
data, overlay function set and program may be transmitted interface (46) by the processing unit (40) may be sent 
to the head end (2) or a plurality of head ends for ultimate through the network interface (36) via communication 
transmission to users as further described herein. ^5 medium (32) to a transaction processor (24) located at the 

It is to be understood that a number of unique protocols head end (2), which causes the implementation of the 

may be designed to implement the synchronization of over- overlay functions. It should be understood at this point that 
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communication medium (32) and communication medium 
(30) may be the same medium or they me be different. For 
example, communication medium (30) may be a sateUite 
signal, while communication medium (32) may be electrical 
wire, or alternatively, a two-way optical fiber may be used 5 
as the sole communication medium between the head end 
(2) and a plurality of remote user locations. The transaction 
processor may be coupled to Internet servers (28), databases 
(26), computer networks and other components to facilitate 
the implementation of the various overlay functions selected 
by users. For example, if an overlay function allows the user 
to connect to a world-wide web site containing information 
related to an actionable event in the programming being 
presented, the transaction processor may be connected to an 
Internet server to allow the user to access the world-wide ^5 
web site. As another example, if an overlay function allows 
the user to request to be reminded when a future program is 
about to begin, the transaction processor may be connected 
to a computer system set up to store unique user addresses 
and program reminder information to be sent to such 20 
addresses. At block (84), if the user did not select an overlay 
function, the processing unit (40) determines whether the 
current actionable event has ended. If the actionable event 
has not ended, the processing unit (40) continues to deter- 
mine whether the user has selected an overlay function. If 25 
the actionable event has ended, the processing unit (40) 
revSets and awaits the next actionable event. 

In a second embodiment of the present invention, the 
synchronization of the interface data and the overlay func- 
tion set with the associated actionable event occurs in the 30 
set-top box (34) instead of in the head end (2), In such an 
embodiment, referring again to FIG. 1, the processing unit 
(12) may determine from data received prior to receiving the 
sequentially coded program whether overlay function sets 
and interface data are available for the program (i.e., 35 
whether there are one or more actionable events occurring in 
the programming). If overlay function sets and interface data 
are available, the processing unit (12) retrieves them from 
memory (16 and 18) and sends them to the set-top box (34) 
at some time prior to the transmission of the program to the 40 
set-top box (34). The processing unit (12) also retrieves 
synchronization data from memory (14) which, as further 
described below, will allow the set-top box (34) to synchro- 
nize overlay function sets and interface data with actionable 
events based upon sequential code data. It is to be under- 45 
stood that the synchronization data, overlay function sets 
and interface data may be stored with the associated pro- 
gram (in signal storage (6) or in storage at the location from 
which satellite transmitted programs are transmitted) and 
transmitted immediately prior to the transmission of the 50 
program to the switcher (8) and subsequently to the pro- 
cessing unit (12) and on to the set-top box (34). 

The synchronization data, overlay function set and inter- 
face data, and the sequentially coded program, are received 
at the set-top box (34) by network interface (36) coupled to 55 
decoder (38). The decoder (38) sends the synchronization 
data, interface data and overlay function data to a processing 
unit (40) which stores; them in memory (44). The processing 
unit (40) performs the same process detailed in FIG. 4 as 
described with respect to the first embodiment, whereby the 60 
processing unit (40) is programmed to read the sequential 
code data associated with the program and reference it to the 
synchronization data stored in memory (44) to determine 
whether overlay function sets and interface data exist for the 
sequential code data read by the processing unit (40) (i.e., 65 
whether there are actionable events occurring in the 
programming). If overlay function sets and interface data 



,908 

10 

exist for the sequential code data read by the processing unit 
(40), the processing unit retrieves the overlay function sets 
and interface data from memory (44). The processing unit 
(40) reads the interface data and implements the user inter- 
face in accordance with the instructions contained within the 
interface data, including sending visual, audible, textual, and 
other material to the signal combiner (50) for presentation to 
the user in synchronization with the actionable event being 
sent to the audiovisual display unit (52). The processing unit 
(40) also reads the overlay function set and enables overlay 
functions in synchronization with the actionable event being 
sent to the audiovisual display unit (52) so that the user is 
able to implement the various overlay functions. 

In a third embodiment of the present invention, it should 
be appreciated that a program and related interface data and 
overlay function sets may be stored in transportable media 
such that actionable events in the programming are synchro- 
nized with associated interface data and overlay function 
sets prior to presentation to a user. In such an embodiment, 
the synchronization process described in the first embodi- 
ment and shown in FIG. 1 as occurring in the head end (2) 
may be performed as described, except that the combined 
signal which is output by the signal combiner (10) may be 
stored in storage (20) for later transmission from the head 
end (2) as already described for the first embodiment, or for 
transfer to copies of transportable media, such as videotape, 
compact disc, DVD or other formats. Alternatively, the 
combined signal which is output by the signal combiner (10) 
may be transmitted to the remote user location, as described 
in the first embodiment, and then stored in transportable 
media by an appropriate recording/playback device (54) 
coupled to decoder (38) through external device interface 
(56). As shown in FIG. 2, during playback from transport- 
able media by recording/playback device (54), the combined 
signal is transmitted through external device interface (56) 
to decoder (38) and is processed as described in the first 
embodiment presented herein. Although the transmission of 
the program would originate in the user's home or other 
remote location from the recording/playback device (54), it 
would still be possible for one or more of the overlay 
functions selected by the user to be performed at the head 
end (2) or other remote location instead of in the set-top box 
(34). Signals received from the user at the control interface 
(46) by the processing unit may be sent through the network 
interface (36) via communication medium (32) to a trans- 
action processor (24) located at the head end (2) or other 
remote location, which would implement the overlay func- 
tions. 

In yet another embodiment of the present invention, 
actionable events for a program may be selected, and 
interface data and overlay function sets may be created or 
selected for those actionable events and made available to 
users, during the transmission of the program. In this 
embodiment, a human operator using a general purpose 
computer located at a head end location (2) may select 
overiay functions and create an interface, or select a preex- 
isting interface, for a given event that the operator wishes to 
make actionable for users. After selecting the overlay func- 
tions and the interface for the current event which the 
operator wishes to make actionable, the operator instructs 
the processing unit (12) in the head end to immediately send 
the overlay function set and interface data to the signal 
combiner (10) to be synchronized with the event being 
transmitted to users at the time. In this embodiment, it is not 
necessary for the program to be sequentially coded in 
advance of transmission since the operator determines when 
the overlay function sets and interface data should be sent to 
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users. Upon receipt at the set-top box (34), the program, the 
interface data and the overlay function sets are handled in 
the same fashion as the first embodiment described herein, 
whereby the overlay function sets and interface data are 
processed by the processing unit (40) and presented to users 5 
immediately. 

There has been described novel methods and apparatus 
for linking real-time data with audiovisual content to enable 
a user to make selections, manipulate data, and to execute 
functions by interacting with the audiovisual content being lo 
shown at the time. It is evident that those skilled in the art 
may now make numerous uses and modifications of and 
departures from the specific embodiments described herein 
without departing from the inventive concepts. 
Consequently, the invention is to be construed as embracing 15 
each and every novel feature and novel combination of 
features present in or possessed by the apparatus and meth- 
ods herein disclosed and limited solely by the spirit and 
scope of the appended claims. 

We claim: 20 

1. A method of distributing audiovisual programming, 
comprising the steps of: 

displaying live programming to a human operator located 
at the head end of a distribution system while distrib- 
uting for display the five programming at substantially 
the same lime to a plurality of remotely located users; 

automatically detecting one of a plurality of pre-selected 
visual objects present within the live programming as 
an actionable event, during which a user selection may 
be accepted; 

adding data for distribution with the live programming in 
response to the detection of the visual object and in 
response to an operator selection of another actionable 
event made by the human operator; 35 

adding synchronization data for distribution with the live 
programming, delineating a time period within the live 
programming display during which a user selection 
may be accepted; 

adding overlay function data for distribution with the live 
programming, controlling the distribution of alternate 
programming to be displayed in response to an indi- 
vidual iiser making a user selection in response to the 
actionable event; 

adding interface data for distribution with the live 
programming, providing indication to each of the plu- 
rality of users of the opportunity to make a user 
selection; 

recognizing a user selection by one of the plurahty of 
users of the opportunity to make a user selection; 

distributing the user selected alternate programming for 
display to the user. 

2. The method of claim 1, wherein the step of adding data 
further comprises the step of: 55 

adding interface data for visually indicating to the user the 
opportunity to make a user selection. 

3. The method of claim 1, wherein the step of adding data 
further comprises the step of: 

adding interface data for visually indicating with an ^0 
animated icon the opportunity to make a user selection. 

4. The method of claim 1, wherein the step of adding data 
further comprises the step of: 
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adding interface data for indicating to the user with audio 
prompting the opportunity to make a user selection. 

5. The method of claim 1, wherein the step of adding data 
further comprises the step of: 

adding interface data for visually indicating with two or 
more selectable screen areas the opportunity to make a 
user selection. 

6. The method of claim 1, wherein the step of adding data 
further comprises the step of: 

adding data for visually distinguishing the automatically 
detected visual object in the live programming being 
displayed to the plurality of users. 

7. The method of claim 1, wherein the step of distributing 
alternate programming is performed without storing infor- 
mation indicating which programming said user was view- 
ing prior to making said user selection. 

8. The method of claim 1, further comprising the steps of: 
displaying said alternate programming to a second human 

operator; 

adding second data to said alternate programming in 

response to a second operator selection of a second 

actionable event; 
recognizing a second user selection made by said user in 

response to the second actionable event; and 
distributing second alternate programming for display to 

said user. 

9. A system for distributing audiovisual programming, 
comprising: 

a signal source providing live programming, displayed to 
a human operator located at the head end of a distri- 
bution system while distributed at substantially the 
same lime to a plurality of remotely located users; 

a processing unit located at the head end, coupled to the 
signal source, which is programmed to: 

i) detect one of a plurality of pre-selected visual objects 
present within the live programming as an actionable 
event, during which a user selection may be 
accepted; and 

ii) add data for distribution with the live programming 
in response to the detection of the visual object and 
in response to an operator selection of another 
actionable event made by the human operator, the 
added data including 

a) synchronization data delineating a time period 
within the live programming display during which 
a user selection may be accepted, 

b) overlay function data controlling the distribution 
of alternate programming to be displayed in 
response to an individual user making a user 
selection in response to the actionable event, and 

c) interface data providing indication to each of the 
plurality of users of the opportunity to make a user 
selection; 

transmission equipment distributing the live program- 
ming for display to the plurality of users; 

a set-top box recognizing a user selection in response to 
the selected actionable event; and 

a transaction processor distributing the alternate program- 
ming to the user in response to the user selection. 

***** 
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